Static Compression for Dynamic Texts

Two problems arise when semi-static word-based compression methods are applied to large texts, such as those stored in information retrieval systems. First, the space required for the model during decoding can become very large. Second, the need to handle document insertions means that the collection must be periodically recompressed if compression eeciency is to be maintained. Here we show that with careful management the impact of both of these drawbacks can be minimised. Experiments with a word-based model and over 500 Mb of text show that compression rates can be retained even in the face of severe memory limitations on the decoder, and in the face of signiicant expansion in the size of the text itself. The use of a word-based zero-order compression model to represent English text has been considered by several authors 2, 6, 7, 15]. It is particularly appropriate for compressing full-text document collections, an application in which very large quantities of text are stored, but individual documents must be independently decodable 1]. In this case a semi-static approach that makes a preliminary pass over the text before compressing it in a second pass is appropriate, and when the word-based model is coupled with static Huuman coding, it yields the three necessary properties: good compression, fast decoding, and random access 10]. There are, however, two drawbacks to the use of this combination. First, a great deal of decode-time memory space might be required to store the tokens of the compression alphabet. Experience has shown that the number of distinct \words" in a text grows as an almost linear function of its size, without the tailing-oo eeect often predicted. If for no other reason, this happens because random spelling mistakes occur at a reasonably constant rate, and these are all regarded as novel words by the compression system. As part of the international trec information retrieval experiment we have been dealing with several corpora of English text, each of approximately 500 Mb 5]. One of the collections is several years of articles from the Wall Street Journal, and this 508.15 Mb wsj database uses 289,101 distinct words totalling 2,159,044 characters; and 8,912 distinct non-words requiring in total 77,882 bytes. Allowing a 4-byte string pointer for each word, and ignoring for the moment